What is machine learning?

• Using the computer to learn from data (yes, it is that simple)
• Applies statistics (finding information in data) and computer science (algorithmic design and implementation)
• Not quite equivalent to AI; AI is built on ML
  • Historically speaking, AI dates back to 1950s and aims to emulate human cognition and behavior, while ML originates in finding insights in data
  • The two have converged because ML are the instruments used to approach AI
  • Ex. an AI which “sees” subjects in an image, is in fact deploying ML methods that parse the image into regions/forms of interest, then classify the form into some category, based on what has been seen and categorized before. Roughly speaking, each component of this process corresponds to an ML algorithm.

ML (and statistics) terminology

Often describes statistical concepts with different language, due to separate disciplinary traditions.

Term review: regression and classification

Supervised learning

Supervised learning has a predictive output or target (regression or classification of a variable). A model is fit which predicts (or retrodicts) some \(y\) from one or more \(x\).

• Regression (esp. linear and logistic)
• \(K\)-Nearest Neighbors classifier
• Decision trees
• Support vector machines (SVM)
• Naive Bayes classifier
• Linear discriminant analysis (LDA)
• Neural networks (incl. deep learning)

Regression

\(K\)-Nearest Neighbors classifier

• An observation of interest is compared to its \(K\) nearest neighbors in \(p\)-dimensional space.
  • \(K\) is the tuning parameter, chosen by the analyst
  • The observation is classified according to the majority of its neighbors

Pt is an observation of unknown category, which we would like to classify. The K nearest neighbors (K = 7 here) are found using a distance function, and the majority class of those neighbors is assigned to Pt. The result would be Class ii in this case. Image source: Atallah, Badawy, and El-Sayed 2019

Decision trees

• We stratify the predictor space using splitting rules
  • Recursive binary splitting:
    select the \(X\) most correlated with \(Y\) \(\rightarrow\)
    find a good “cut point” (decision boundary) to split the data, according to \(Y\) \(\rightarrow\)
    in the two resulting bins, the process is repeated, and so on, until a stop condition is reached.
• May be used for classification but also regression
• Very explainable, but “typically are not competitive with the best supervised learning methods” in terms of accuracy (James et al. 2021); also not very robust on their own
  • Methods like random forests, bagging, and boosting extend the decision tree algorithm and address issues of accuracy and robustness

Support vector machines (SVM)

Neural networks

• A neural network takes \(X_1…X_p\) variables and builds a nonlinear \(f(X)\) to predict the response \(Y\) (James et al. 2021)
• Derived features are computed and utilized by hidden layers, between the input and output; nodes have different rules for activating
• When there is more than one hidden layer, it is called deep learning
• This is the black box method: very sophisticated, but little or no explainability for an individual given result

Model assessment

• There are many tools and possible implementations of supervised methods
• “More data” in the model isn’t automatically better (e.g., can create bias with highly correlated variables)
• Validation
  • Prior to analysis, data are split into training and test sets
  • The model is fit on the training data
  • Then the test data are submitted to the model, and the test \(Y\) is compared to the model’s \(\hat{Y}\) prediction—a measure of accuracy (was the model right?)
• Cross-validation: a more sophisticated take
• Feature selection is also an important activity, informed by validation as well as techniques such as lasso regression

Principal Component Analysis (PCA)

• \(X_1, X_2,…,X_p\) features are reduced to a small number of “principal components”
  • \(Z_1\), the first principal component, accounts for most of the variation in the data
  • \(Z_2\) accounts for most of the remaining variation
• Example of dimensionality reduction 🤏
• Useful for:
  • understanding
  • deriving variables for supervised methods
  • visualizing 3+ variables in a 2D space
  • imputation (guessing empty values)
• Similar to Linear Discriminant Analysis (LDA), a supervised method which incorporates dimensionality reduction among predictors

\(K\)-Means Clustering

• A number \(K\) of previously unknown clusters are identified; or: we partition the data into \(K\) clusters
  • \(K\) is chosen by the analyst
  • Clusters aren’t completely random, but based on similarities in the data
  • Assign every observation a number 1 through \(K\) \(\rightarrow\)
    calculate each cluster centroid \(\rightarrow\)
    (re)assign each observation to the closest centroid \(\rightarrow\)
    continue calculating and reassigning until movement stops
  • Total within-cluster variation is minimized
• Not a predictive classifier!
• Useful for:
  • Exploration
  • Identifying potential subpopulations